Hi there! I tried this PR (applied the changes on top of the last commit, to use MLA + FA) on DeepSeek V3 0324, but I noticed less PP performance. I think it doesn't saturates the PCI-E for the main GPU when using the flag vs without, which results in less PP t/s.

Command to run was

./llama-server -m '/GGUFs/DeepSeek-V3-0324-UD-Q2_K_XL-merged.gguf' -c 16384 --no-mmap --no-warmup -v -ngl 99 --override-tensor 'blk\.([0-7])\..*_exps\.=CUDA0' --override-tensor 'blk\.([8-9]|1[0-1])\..*_exps\.=CUDA1' --override-tensor 'blk\.(1[2-6])\..*_exps\.=CUDA2' --override-tensor 'blk\.(1[7-9]|2[0-6])\..*_exps\.=CUDA3' -fa --override-tensor 'blk\..*_exps\.=CPU' -mg 0 --ubatch-size 1024 --disable-op-offload

RX/TX usage without the flag (GPU 0 gets saturated) while doing PP

RX/TX usage with the flag while doing PP

So PP t/s go from 66 t/s to 26 t/s

No flag:

prompt eval time =   35950.29 ms /  3218 tokens (   11.17 ms per token,    89.51 tokens per second)
       eval time =   44338.15 ms /   380 tokens (  116.68 ms per token,     8.57 tokens per second)

Flag:

prompt eval time =  122421.67 ms /  3218 tokens (   38.04 ms per token,    26.29 tokens per second)
       eval time =   49715.68 ms /   440 tokens (  112.99 ms per token,     8.85 tokens per second)

Maybe there is an incompatible flag I'm using?

Using the flag should place way more load on the CPU, and whether that's beneficial at all should be highly specific to model offload params and hardware config.

For now I personally only tested llama 4 and qwen 3 with a relatively performant CPU (7970X) and a single relatively weak GPU (W7900), with a simple exps=CPU ot config, so YMMV. There's already a huge variance in terms of perf uplift in the 2 models I tested.

./build/bin/llama-bench -m ~/models/llama4-400b-hybrid-q8_0-q4_0.gguf -ngl 999 -fa 1 -ctk q8_0 -ctv q8_0 -ot 'exps=CPU' -mmp 0 --disable-op-offload 1,0 -n 0
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: AMD Radeon PRO W7900 Dual Slot , gfx1100 (0x1100), VMM: no, Wave Size: 32

./build/bin/llama-bench -m ~/models/qwen3-235b-a22b-q8_0-q4_0-hybrid.gguf -ngl 999 -fa 1 -ctk q8_0 -ctv q8_0 -ot 'exps=CPU' -mmp 0 --disable-op-offload 1,0 -n 0
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: AMD Radeon PRO W7900 Dual Slot , gfx1100 (0x1100), VMM: no, Wave Size: 32

Ah then that could be the reason then. I have 192GB RAM on a Ryzen 7 7800X3D, which is a consumer CPU so it is pretty weak for these tasks.

Could we not just parameterise the fixed 32 batch size threshold to offload? This would still let you disable offloading by setting to a very large value, but also allow more nuanced setting?

A setting to control the minimum batch size would need to be per-backend, and configured via an environment variable.

A setting to control the minimum batch size would need to be per-backend, and configured via an environment variable.

Ah, sorry I forgot the 32 limit I was thinking of is CUDA specific.

FYI: using --no-op-offload on Dell R740 with two Nvidia 4090D 48G slows down prompt processing a lot
with the flag

prompt eval time =   78715.84 ms /  5480 tokens (   14.36 ms per token,    69.62 tokens per second)
eval time =   36350.14 ms /   195 tokens (  186.41 ms per token,     5.36 tokens per second)
total time =  115065.99 ms /  5675 tokens

without the flag

prompt eval time =  268849.41 ms / 75130 tokens (    3.58 ms per token,   279.45 tokens per second)
eval time =    6960.09 ms /    39 tokens (  178.46 ms per token,     5.60 tokens per second)
total time =  275809.51 ms / 75169 tokens